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1. Computational experiments should he recomputable for all time 

2. Recomputation of recomputable experiments should he very easy 

3. Tools and repositories can help recomputation become standard 
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Vh ' A. It should be easier to make experiments recomputable than not to 

.^ \ 5. The only way to ensure recomputability is to provide virtual machines 

CN ■ 6. Runtime performance is a secondary issue 

Replication of scientific experiments is critical to the advance of scienceu Unfortunately, 
the discipline of Computer Science has never treated replication seriously, even though com- 
puters are very good at doing the same thing over and over again. Not only are experiments 
rarely replicated, they are rarely even replicable in a meaningful way. Scientists are being 
encouraged to make their source code available [13], but this is only a small step. Even in 
the happy event that source code can be built and run successfully, running code is a long 
way away from being able to replicate the experiment that code was used for. 

>• '■ 
^. I I propose that the discipline of Computer Science must embrace replication of experiments 

t~~. ' as standard practice. I propose that the only credible technique to make experiments truly 

^O , replicable is to provide copies of virtual machines in which the experiments are validated to 

run. I propose that tools and repositories should be made available to make this happen. I 

^r . propose to be one of those who makes it happen. 

C^ I I am using the word 'recomputation' to mean the replication of computational experiments. 

This is the 'recomputation manifesto', not the 'replication manifesto'. The word 'replication' 
has a well-established technical meaning in computing. I am adding a technical meaning 
^ ' to an existing word|j but the new meaning adds only a nuance and also serves usefully to 

5h ■ distinguish replication of computational experiments from other types of experiment. 



1. Computational experiments should be recomputable for all time. 

Why recomputable? And why for all time? 

The necessity for recomputation is simple. An experiment is used to form scientific conclu- 
sions, and those conclusions are never final. They may always be subject to question, so 
experimental work may need to be repeated. I would even argue that the less experiments 
are recomputed, the more important it is that they be recomputable. The reason is that 
flaws in an experiment that is frequently recomputed are likely to come to light, but not 



^As well as being critical, it is a timely issue, witness this in Nature, 3 April 2013 17 . 
■^The word 'recomputation' is older than the USA, reported by the Oxford English Dictionary in 1766. It 
does have some technical meanings, e.g. [18], but none as widespread as those for 'replication'. 
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SO if the experiment is rarely recomputed. The more rarely an experiment is repeated, 
the longer the time lapse until it is, and the less likely a new researcher will be able to 
recompute it easily without the original experiment being deliberately made recomputable. 
If the original experiment is flawed in some way, misleading results can lie uncorrected in 
the literature for years [1]. 

Should experiments be recomputable for all time? Yes. Computer Science is unique in 
having this ability. Imagine if physicists had the opportunity to look through Galileo's 
telescope at Jupiter's moons, but had thrown them away on the basis that a description of 
how to make it was available. There is no reasonable limit to what is the useful life of an 
experiment is, so the simplest answer is to have no limit. Moore's law, while it lasts, makes it 
technically feasible to store experiments forever. It's affordable to store experiments: if cost 
per GB decreases exponentially, a fixed sum is enough to store each GB in perpetuity. It is 
even affordable to rerun all computational experiments in history regularly. If researchers 
donated 10% of CPU resources for new experiments to recomputing old ones, all 5 year old 
experiments could be rerun each year|3 

2. Recomputation of recomputable experiments should be very easy 

Recomputation should be very easy. Ideally, very easy means clicking a button. This will 
not always be possible, but we should make it as easy as we possibly can for experimenters 
to reproduce their own experiments and those of other researchers. A lovely example of 
making it easy to rerun existing experiments is provided by the website [ruiiinycode . org 



which experimenters can use to provide a simple interface to test their code on different 
inputs [19]. 



An important point about this manifesto is that by recomputation I focus mainly on the 
exact replication of an experiment. No other choice is available since (as I argue below) 
we have to duplicate the exact experimental conditions to ensure that recomputation is 
possible. Drummond argues powerfully that this - which he calls 'replicability' - is the 
poor cousin of 'reproducibility', where experiments are reproduced with changes to factors 
believed to be insignificant [7j. There is no question that key advances in science should be 
reproducible in Drummond's richer sense, but neither do I accept that "the impoverished 
version, replicability, is one not worth having" [7]. To take a non-computational example, 
it would be wonderful if the original experiments on cold fusion could be replicated exactly. 
If there was some technical flaw with the experiments, it could be discovered, or if not the 
effects that led to the apparent conclusion that cold fusion existed could be investigated. 
Once we have ensured that experiments can be recomputed exactly, we can have the luxury 
of seeking to ensure that experiments can be reproduced in richer ways. 

3. Tools and repositories can help recomputation become standard 

I draw an analogy with version control tools. Learning and using subversion, git, or mercu- 
rial adds complication to one's life. But the benefits are enormous, and without question it 
is easier overall to build and maintain code using source code control systems than without. 
Similarly, repositories such as github or bitbucket greatly enhance development and shar- 
ing of code. Once we have the appropriate tools and repositories we in Computer Science 
will have an enormous advantage over other sciences. Many sciences have massive online 
repositories, of course leveraging advances in Computer Science to their benefit. We can 



^Assuming computing power doubles every 18 months, CPU resources increase by 10 times every 5 years. 
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be unique in having massive online repositories of fully realised experiments, not just the 
data resulting from experiinents. We should research and make available tools and repos- 
itories for the recomputation of scientific experiments. Sadly, tools for recomputation are 
relatively lacking to date, although there are promising signs. In a single PhD thesis, Philip 
Guo introduced Burrito, a tool for electronic lab notebooks, and CDE, an automatic pack- 
aging tool for experiments in linux [11]. Other important recent tools include "verifiable 
computational results" [8], "Sumatra" [6] and HAL [14j . As well as runmycode.org, other 
repositories helping to make experiments replicable are myExperiment . org [16j, SHARE 



[lOj . and the specialist journal "Image Processing Online" ( [www, ipol. imp in which each 
article is accompanied by runnable code. 

4. It should be easier to make experiments recomputable than not to 

How can it be easier to make new experiments recomputable than not to bother? I draw 
your attention to Jon Claerbout's words [3]: 

"It's not really for the benefit of other people. Experience shows the principal 
beneficiary of reproducible research is you the author yourself. " 

I've been doing major computational experiments for 20 years: some of them I am rather 
proud of. But whenever I start on a new set of experiments, my heart sinks a little. I expect 
the pain of getting everything set up, and then I expect the difficulty or impossibility of 
reproducing my experiments if, say, the paper is rejected and I need to rerun them for a 
later submission. We should fix this. Tools should be available to make it easier to run 
experiments, encourage good experimental practice, and simultaneously make them recom- 
putable. An experienced experimenter's heart should sink no more than an experienced 
programmer when starting a new program. Just as programmers have comfortable devel- 
opment environments they can use, experimenters should have a range of environments to 
develop experiments in. Repositories of past experiments - one's own and other people's - 
should act like software libraries to aid rapid development of new experiments. 

5. The only way to ensure recomputability is to provide virtual machines 

While the other points of the manifesto state how I think the world should be, this point is 
an empirical statement about how the world is. There are two reasons that virtual machines 
are - at least for now - the only way to go. The first is the almost universal (I suspect) 
experience among anyone who has built software and especially tried to rebuild it later. A 
build of the resulting program (and hence experiment) can fail due to arbitrary changes 
in the machine being used. This applies even where the machine is physically the same 
one that was used months ago to run an experiment the first time. An innocent software 
update, to a package not obviously used by the build software, can cause disaster. If the 
machine is not the same one or a clone thereof, all bets are off. The only way to ensure an 
experiment is recomputable is to make available a virtual machine identical to one which 
was tested and worked when the experiment was originally run. The second reason is that 
the available computers and operating systems change over time. It may be true that code 
you make available today can be built with only minor pain by many people on current 
computers. That is unlikely to be true in 5 years, and hardly credible in 20. 

A classic example illustrates both these reasons. SHRDLU is one of the most famous 
programs in the history of AI [20J. The complete source code is available, but cannot be 
run. The physical machine and OS it ran on can be emulated, but we do not have the exact 
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machine state that enabled the program to run. Amongst other issues, Terry Winograd 
changed his own copy of Lisp, changes that are now lost [5j. What we need - and should 
provide for new experiments - is an exact virtual machine in which the experiment worked. 
There are isolated cases of researchers providing virtual machines for recomputation, such 
as Brown [21 |3j and SHARE [TO]. This should be the standard way Computer Scientists 
do business. Howe makes this case much more extensively, giving 13 reasons why virtual 
machines in the cloud can improve reproducibility 



This does not mean that all uploads and downloads to repositories should be of full virtual 
machines. For ease of use, as well as reducing bandwidth, it's important that tools provide 
as many ways as possible of allowing experiments to be uploaded and downloaded. 

6. Runtime performance is a secondary issue 

If a measure is deterministic, e.g. the data structure resulting from a deterministic algo- 
rithm, it should be identical every time the experiment is recomputed. We cannot guarantee 
the same results in non-deterministic measures such as cpu time. By using virtual machines, 
we may obtain different results as are seen on a physical machine, and future results on 
virtual machines may differ from current ones. Even where serious efforts are made, there is 
no guarantee that reproducible run-time results can be obtained. For example, even though 
max-clique researchers have a standard methodology for cross-referencing cpu times, Prosser 
has shown that these results cannot be relied on even approximately fl5j. 

We must tackle the easier problems first. Ensuring recomputation of experiments is not easy, 
but it is vital. Obtaining meaningfully replicable cpu time comparisons requires significant 
additional research - if it is even possible. By allowing recomputation of experiments, we 
allow researchers to do them in a variety of environments, discovering if conclusions about 
run times are universal or contingent. The crucial thing is to preserve scientific experiments. 
It's unarguable that if we can't recompute an experiment at all, we can't preserve run time 
performance. 

The recomputation.org Mission 



To help make recomputation a reality, I am starting http://recomputation.org as one 
repository for recomputable experiments, and starting to work on the tools to use on the 
site. Based on my arguments above I have the following plans. The overriding goal is the 
following mission statement: 

// we can compute your experiment now, anyone can recompute it 20 years 
from now 

The site |recomputation . org| is brand new and (as I write) holds zero experiments. But 
the following are the principles which we will use in developing it to deliver the mission 
statement. 



1. |recomputation.org| will make available virtual machines or equivalent technology 



to allow exact recomputation of lodged experiments. 



2. recomputation. org| will make its best efforts to ensure that all experiments which 



it believes to be recomputable will remain recomputable for all time. 



3. recomputation. org] will always be free to those lodging bona-fide scientific exper- 



iments and to those obtaining past experiments, provided that: all aspects of the 
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experiments are freely available; the experimenter's contributions are open source; 
and fees are not charged for the related scientific publicationO 



4. Irecomputation. org|will provide its code and tools using an appropriate open source 



licence, including server-side code. 

5. Irecomputation. org| will serve as a testbed for scientific research into issues such as 
experimental techniques and methodologies. 



Vote Recomputation 

The recomputation manifesto is similar in spirit to the 'Science Code Manifesto' but ad- 
dresses an orthogonal issue. The Science Code Manifesto, sciencecodemanif est o . org| 
demands that code used in scientific publications should be made available in an open way 
for the useful lifetime of the publication. But this is neither a necessary condition for 
recomputation nor sufficient for it. Closed source experiments can be recomputed if the 
appropriate environment is provided - e.g. a suitable virtual machine containing the neces- 
sary binaries but not source. Arguably recomputability is more important for closed source 
than for open source. But it must not be thought that open source is sufficient for recom- 
putability. The only guarantee of recomputability is the exact environment being available. 
I do endorse the Science Code Manifesto, and initial efforts at http://recoinputation.org' 
will be focussed on open source efforts only. As well as the scientific desirability of open 
source, it avoids one form of potential licencing problems in recomputation. 

A manifesto is a call that people reading it should vote for your point of view. Don't vote 
with a signature or a petition. Vote by making your computational experiments recom- 
putable. Do it at http://recomputation.org, or at your own web site, or at another 
repository. But make your experiments recomputable. 
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